Define special methods for ndarray and add more extensive tests. #10

shoyer · 2018-02-20T01:01:55Z

No description provided.

shoyer · 2018-02-20T01:02:43Z

tests/test_simple.py

+array_scalar = np.array(1)
+int(array_scalar)
+float(array_scalar)
+complex(array_scalar)


mypy current errors on this line: tests/test_simple.py:41: error: No overload variant of "complex" matches argument types [numpy.ndarray]

Is this a mypy bug or am I doing something wrong?

Ah, this is a bug it would appear. Since ndarray subscribes to the SupportsComplex protocol (sorry, I missed that initially), I see no reason this shouldn't work.

Reported at python/mypy#4592

shoyer · 2018-02-28T19:29:04Z

@eric-wieser Want to take a look?

shoyer · 2018-03-06T00:01:51Z

@alanhdu any thoughts?

eric-wieser · 2018-03-06T00:22:12Z

numpy/__init__.pyi

+    def __float__(self) -> float: ...
+    def __complex__(self) -> complex: ...
+    def __oct__(self) -> str: ...
+    def __hex__(self) -> str: ...


These should be conditional on the python version. I think mypy supports that?

Yes, it supports limited version checking with sys.version_info.

eric-wieser · 2018-03-06T00:26:43Z

tests/test_simple.py

+len(array)
+str(array)
+array + 1
+-array


Do you want to test all the binary operators here too?

Yeah, that's probably a good idea.

eric-wieser · 2018-03-06T00:27:45Z

numpy/__init__.pyi

    def __getattr__(self, name) -> Any: ...


 def array(
    object: object,
-    dtype: dtype = ...,
+    dtype: _dtype_class = ...,


_dtype_class is "things which can be converted to a dtype"?

No, it's literally the dtype class. This should be relaxed to something more flexible, e.g., Union[Type[np.generic], str, dtype] (plus probably some other types for structured dtypes).

eric-wieser · 2018-03-06T08:24:05Z

numpy/__init__.pyi

-    def __hex__(self) -> str: ...
+    if sys.version_info.major < 3:
+        def __oct__(self) -> str: ...
+        def __hex__(self) -> str: ...
    def __nonzero__(self) -> bool: ...


This is called bool on py3

eric-wieser · 2018-03-06T08:27:40Z

numpy/__init__.pyi

@@ -83,7 +109,8 @@ class dtype:
    def type(self) -> builtins.type: ...


-_dtype_class = dtype      # for ndarray type
+_DtypeLike = Union[dtype, _ConvertibleToDtype]
+_Dtype = dtype  # to avoid name conflicts with ndarray.dtype


It would be great if we could have a "strict-typing" mode that narrowed the scope of what dtype-like or array-like means. I wonder how that could be achieved...

Yes, I'd love see this.

eric-wieser · 2018-03-06T08:28:26Z

numpy/__init__.pyi

+# are typed as Any.
+# Refernce: https://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
+_ConvertibleToDtype = Union[
+    type,  # TODO: enumerate np.generic types and Python scalars


Does Type[np.generic] work here?

It would, but we haven't defined the scalar type hierarchy yet.

shoyer · 2018-03-06T17:07:12Z

numpy/__init__.pyi

+# definitions, any nested fields are required to be castable to a dtype object
+# are typed as Any.
+# Reference: https://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
+_ConvertibleToDtype = Union[


Apparently (with mypy) I can actually include a forward reference to dtype in this top-level type alias. If that's a standard feature for pyi files, then I suppose we should make use of it rather than defining the separate _DtypeLike alias below.

~~Nevermind, this doesn't work.~~ It does work.

Just pushed a commit implementing this.

eric-wieser · 2018-03-06T08:38:12Z

numpy/__init__.pyi

+    # TODO: add a protocol for anything with a dtype attribute
+    str,
+    Tuple[Any, int],
+    Tuple[Any, _Shape],


Could combine these with a ShapeLike - which is consistent with how np.zeros(shape_like) and a bunch of other functions work.

eric-wieser · 2018-03-06T17:24:47Z

numpy/__init__.pyi

+    List[Union[Tuple[Union[str, Tuple[str, str]], Any],
+               Tuple[Union[str, Tuple[str, str]], Any, _Shape]]],
+    Dict[str, Any],
+    Tuple[Any, Any]]


I think each of these could do with a comment giving an example of the case they aim to cover

eric-wieser · 2018-03-06T18:11:22Z

numpy/__init__.pyi

+    # (flexible_dtype, itemsize)
+    Tuple[Any, int],
+    # (fixed_dtype, shape)
+    Tuple[Any, _Shape],


Note that (int, 2) is shorthand for (int, (2,)), so this the int case covers more than you might expect

OK, in that case I'm going to use _ShapeLike = Union[int, Tuple[int, ...]] for clarity.

eric-wieser · 2018-03-06T18:12:45Z

numpy/__init__.pyi


 from numpy.core._internal import _ctypes

 _Shape = Tuple[int, ...]

+# Anything that can be coerced into numpy.dtype. To avoid recursive
+# definitions, any nested fields are required to be castable to a dtype object


Perhaps do _DtypeLike_nested = Any # TODO: wait for support for recursive types, and then use it in place of Any below?

eric-wieser · 2018-03-07T09:06:57Z

numpy/__init__.pyi

+        def __nonzero__(self) -> bool: ...
+    else:
+        def __bool__(self) -> bool: ...
+    def __bytes__(self) -> bytes: ...


I imagine this is python 3 only?

Also __unicode__, __str__, and __repr__ are missing

alanhdu · 2018-03-07T18:24:33Z

numpy/__init__.pyi

+    # {'names': ..., 'formats': ..., 'offsets': ..., 'titles': ...,
+    #  'itemsize': ...} or {'field1': ..., 'field2': ..., ...}
+    # TODO: use TypedDict when/if it's officially supported
+    Dict[str, Any],


Is this really Any? I think there could be:

Union[ Dict[str, Sequence[Union[int, str, _DtypeLike]]] # for {"names": ..., "formats": ...} Dict[str, Union[Tuple[str, int], Tuple[_DtypeLike, int]]] # for {"field": ....} ]

I think that should be

# for {"names": ..., "formats": ...} Dict[str, Union[ Sequence[str], #names Sequence[_DtypeLike], # formats Sequence[int], # offsets int, # itemsize ]], Dict[str, Tuple[_DtypeLike, int]] # for {"field": ....}

This is probably dangerous because dict is an invariant type (so, for example, Dict[str, int] wouldn't be compatible with this big union type). Can you use Mapping instead?

We could use Mapping, but it would be inaccurate. NumPy does actually insist on dict:

In [13]: od = collections.OrderedDict(x=(float, 1), y=(int, 2)) In [14]: np.dtype(od) Out[14]: dtype({'names':['x','y'], 'formats':['<f8','<i8'], 'offsets':[1,2], 'itemsize':10}) In [15]: np.dtype(ChainMap([od])) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-15-03baf5fab980> in <module>() ----> 1 np.dtype(ChainMap([od])) TypeError: data type not understood

We have the same problem with:

List[Union[ Tuple[Union[str, Tuple[str, str]], _DtypeLikeNested], Tuple[Union[str, Tuple[str, str]], _DtypeLikeNested, _ShapeLike]]],

which maybe should be Sequence but NumPy insists on taking a list (or at least, it doesn't take a tuple which is my go-to example of a non-list sequence).

I added the more specific types for Dict, which should at least minimize the invariance issue.

Is it possible to make a dict that is covariant in its argument type? e.g.,

K = TypeVar('V', covariant=True) V = TypeVar('V', covariant=True) def f(x: Dict[K, V]): ...

eric-wieser

Did this need changing? Why?

eric-wieser · 2018-03-08T07:29:14Z

(re the text/Unicode commit)

shoyer · 2018-03-08T15:54:25Z

Flake8 didn't like using unicode on Python 3. I guess I could have used a comment to disable the flake8 check but this was an easy work around.

eric-wieser · 2018-03-10T19:07:06Z

numpy/__init__.pyi

+    def __abs__(self) -> ndarray: ...
+    def __invert__(self) -> ndarray: ...
+
+    # TODO(shoyer): remove when all methods are defined
    def __getattr__(self, name) -> Any: ...


Missing __setattr__ for .dtype and .shape?

dtype and shape are only typed as attributes, not properties, which means they can be set. But perhaps it would indeed be good to overload setters appropriately...

shoyer · 2018-03-13T15:41:25Z

@eric-wieser @alanhdu thanks for the careful reviews!

Define a special methods for ndarray and add more extensive tests.

46519eb

shoyer commented Feb 20, 2018

View reviewed changes

shoyer changed the title ~~Define a special methods for ndarray and add more extensive tests.~~ Define special methods for ndarray and add more extensive tests. Feb 20, 2018

shoyer mentioned this pull request Feb 20, 2018

BUG: SupportsComplex doesn't seem to work python/mypy#4592

Closed

shoyer added 2 commits February 20, 2018 00:15

Disable failing test

105e699

Remove SupportsRound

134bd7a

eric-wieser reviewed Mar 6, 2018

View reviewed changes

shoyer added 5 commits March 5, 2018 23:02

Don't define removed special methods on Python 3

878545a

Add _DtypeLike

8c03e56

Tests for arithmetic

c17efdb

Grammar

5430fb7

Adjust version checking

e9e9523

eric-wieser reviewed Mar 6, 2018

View reviewed changes

shoyer added 3 commits March 6, 2018 08:55

Update mypy requirement and ncomment divmod test

301161c

Add __bool__, update comments

e3d5cdd

Add None to DtypeLike

233ff66

shoyer commented Mar 6, 2018

View reviewed changes

eric-wieser reviewed Mar 6, 2018

View reviewed changes

Remove _ConvertibleToDtype in favor of _DtypeLike

40847a4

eric-wieser reviewed Mar 6, 2018

View reviewed changes

Add comments for _DtypeLike

ff9dd2a

eric-wieser reviewed Mar 6, 2018

View reviewed changes

Add _DtypeLikeNested

9ac0ddc

shoyer mentioned this pull request Mar 6, 2018

Exporting type aliases for DtypeLike, ShapeLike and ArrayLike #13

Closed

eric-wieser reviewed Mar 7, 2018

View reviewed changes

alanhdu reviewed Mar 7, 2018

View reviewed changes

shoyer added 2 commits March 7, 2018 21:23

Replace Dict[str, Any] in _DtypeLike and add missing ndarray methods

6297521

unicode -> Text

087f725

eric-wieser reviewed Mar 8, 2018

View reviewed changes

eric-wieser reviewed Mar 10, 2018

View reviewed changes

Convert dtype, shape and strides into properties with setters

c6e6e95

eric-wieser approved these changes Mar 13, 2018

View reviewed changes

shoyer merged commit 3074ca3 into numpy:master Mar 13, 2018

shoyer deleted the special-methods branch March 13, 2018 15:41

Uh oh!

Define special methods for ndarray and add more extensive tests. #10

Define special methods for ndarray and add more extensive tests. #10

Uh oh!

Conversation

shoyer commented Feb 20, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmatyping Feb 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shoyer commented Feb 28, 2018

Uh oh!

shoyer commented Mar 6, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shoyer Mar 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shoyer Mar 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser left a comment

Choose a reason for hiding this comment

Uh oh!

emmatyping Feb 20, 2018 •

edited

Loading

shoyer Mar 6, 2018 •

edited

Loading

shoyer Mar 6, 2018 •

edited

Loading